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MEDIA COMMUNICATION CONVERTING BURST LOSSES TO ISOLATED 

LOSSES 



TECHNICAL FIELD 
5 Embodiments of the present invention relate to the field of media 

transmission. Specifically, embodiments of the present invention relate to 
streaming packets of media data. 

BACKGROUND 

1 0 Media communication over bit-rate-limited and error-prone channels, 

such as packet networks and wireless links, requires both high compression 
and high error resilience. Important applications within this context include 
streaming media over the Internet, and streaming wireless media to 
handheld devices with wireless local area networks (LANs) or with emerging 

15 Third Generation (3G) cellular systems. To achieve high compression, most 
current media compression systems, such as Motion Picture Experts Group 
(MPEG)-1/2/4 and H.261/3/L, employ motion-compensated prediction 
between frames to reduce the temporal redundancy, a spatial transform to 
reduce the spatial redundancy, and entropy encoding to produce the 

20 compressed bitstream. These algorithms provide significant compression; 
however, the compressed signal is much more vulnerable to packet losses 
when transmitted over error-prone channels such as the Internet. One 
problem that afflicts media compressed with these encoders is error 
propagation. Specifically, although the inter-frame prediction provides 

25 significant compression, the decoder depends on the reference frame used 
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by the encoder in order to perform correct decoding. A channel error that, for 
example, results in loss of this reference frame can cause the reconstructed 
frame at the decoder to be distorted, which in turn can lead to significant 
error propagation to subsequent frames. 

5 

Prior Art Figure 1 is a block diagram 100 showing an example of a 
typical system for transmitting media in accordance with conventional art. In 
the example shown, media data 110 enters an encoder/packetizer 120 
where it is compressed, encoded and packetized for transmission. The 
1 0 packets that result can be stored in a storage medium 1 30 for later 
transmission over channel 150 as scheduled by scheduler 140. 



Scheduler 140 of Figure 1 schedules the packets for transmission at 
the appropriate time and transmits them sequentially, in numerical order, to a 
15 destined receiver/decoder 160. Receiver/decoder 160 decodes and 
decompresses the media and makes it available to a viewer over an 
appropriate viewer medium. 



Prior Art Figure 2A illustrates a stream of packets 200a as transmitted 
20 in the conventional art by scheduler 140 of Figure 1 . Each numbered square 
represents a packet and the numbers represent the sequence number of 
each packet. Thus, it is shown that the conventional stream of media 
packets is sequenced in numerical order. 
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One problem with the conventional art occurs when a number of 
Consecutively transmitted packets are lost for one reason or another. This 
type of loss is known as a "burst" loss, or, in the plural as "bursty" losses. 
Prior Art Figure 2B illustrates a conventional stream of packets 200b 
5 illustrating a burst loss (a loss of several sequential packets) containing 
frames 2, 3 and 4. Frame 1 is used to estimate the three missing frames. 
Since frames 3 and 4 are progressively further from frame 1 , it follows that 
frame 1 provides a progressively worse estimate of the frames 3 and 4. 
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SUMMARY 

Embodiments of the present invention pertain to systems and 
methods for transmitting media packets. The methods include receiving 
media packets in one order and transmitting them in a different order that is 
5 selected according to loss characteristics of a downstream channel. The 
order selected for transmitting the media packets is one that results in 
reduced distortion with respect to the distortion predicted for the first order. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Prior Art Figure 1 is a block diagram showing an example of a typical 
system for transmitting media in accordance with conventional art. 

5 

Prior Art Figure 2A illustrates a stream of packets as transmitted in the 
conventional art. 

Prior Art Figure 2B illustrates a conventional stream of packets 
1 0 illustrating a burst loss. 

Figure 3 is a block diagram of a system for converting burst losses in 
media transmission into isolated losses, according to one embodiment of the 
present invention. 

15 

Figure 4A is an illustration of a block interleave^ in accordance with 
one embodiment of the present invention. 

Figure 4B is an illustration of the effect of a burst loss in a media 
20 transmission utilizing a block interleave^ in accordance with one 
embodiment of the present invention. 

Figure 5A is an illustration of a block interleave^ in accordance with 
one embodiment of the present invention. 

25 
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Figure 5B is an illustration of media transmission utilizing a block 
interleaves in accordance with one embodiment of the present invention. 

Figure 6 is a block diagram of a system for transmitting live media 
5 data for real-time transmission, according to one embodiment of the present 
invention. 

Figure 7 is a block diagram of a system for adapting the scheduling of 
media packets received from one channel and transmitting them according 
10 to the adapted schedule over another channel, in accordance with one 
embodiment of the present invention. 

Figure 8 illustrates a schedule adapter utilizing multiple channels, 
according to one embodiment of the present invention. 

15 

Figure 9 is a flow chart of a method for converting burst losses into 
isolated losses, in accordance with one embodiment of the present 
invention. 
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DETAILED DESCRIPTION 

Reference will now be made in detail to embodiments of the invention, 
examples of which are illustrated in the accompanying drawings. While the 
invention will be described in conjunction with the embodiments, it will be 
5 understood that they are not intended to limit the invention to these 
embodiments. On the contrary, the invention is intended to cover 
alternatives, modifications, and equivalents, which can be included within 
the spirit and scope of the invention as defined by the appended claims. 
Furthermore, in the following detailed description, numerous specific details 
1 0 are set forth in order to provide a thorough understanding of the present 
invention. In other instances, well known methods, procedures, and 
components have not been described in detail so as not to unnecessarily 
obscure aspects of the present invention. 

1 5 Embodiments of the present invention include systems and methods 

for transmitting media packets of data over a channel so as to reduce 
distortion at the receiving end of the transmission. One well-known cause of 
distortion is the burst loss of packets. That is, a number of sequential data 
packets are lost. The present invention, in one embodiment, provides a 

20 method and system with which to convert burst packet losses to effective 

isolated packet losses. The invention is based on the realization that bursty 
channel losses (losses of a number of sequential packets of data packets) 
generally produce larger total mean-square error distortion in streaming data 
than an equivalent number of isolated losses. A packet schedule-reordering 

25 or schedule adapting scheme, such as packet interleaving, for example, is 
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proposed to combat the effect of bursty losses by converting them into 
isolated losses. A schedule adapter, such as an interleaver, for reducing the 
expected total distortion, subject to a delay constraint, can be selected using 
a model that predicts the expected distortion for different packet loss 
5 patterns. 

For purposes of clarity and brevity, the discussion herein references 
an interleaver as the schedule adapter. However, it is appreciated that the 
present invention, in its various embodiments, can be used with other 

1 0 schedule adapters that are able to reorder a stream of data packets. Packet 
interleaving is readily implemented and does not consume additional 
bandwidth. The reordering of the transmission sequence of packets does 
not add any additional bits of data, and thus has no impact on the bandwidth. 
Additionally, interleaving can be used in conjunction with other techniques to 

15 improve media or other data quality, including Forward Error Correction 
(FEC) and error-resilient media coding techniques. 

Figure 3 is a block diagram 300 of a system 305 for converting burst 
losses in video or other media transmission into effective isolated losses at 

20 the receiving end, according to one embodiment of the present invention. 
System 305 includes an encoder/packetizer 320 that receives an incoming 
media stream 310 and encodes and packetizes it, a storage media 330 for 
storing the encoded/packetized media data, interleaver 350, interleaver 
selector 385, channel characteristics estimator/historical data retriever 380 

25 and historical data storage 390. It is appreciated that the blocks in system 
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305 are illustrated separately for clarity of illustration and discussion; 
however, these blocks may not exist as separate entities within system 305. 
It is also appreciated that the blocks of system 305 can be embodied in a 
single device or in a number of devices. In general, system 305 includes 
5 one or more devices that provide the functionality about to be described. 



Furthermore, for purposes of clarity and brevity, the discussion herein 
is for video media data. However, it is appreciated that the present 
invention, in its various embodiments, can be used with other media data 
10 including, but not limited to, image-based data, speech, audio data, graphics 
data, Web page-based data; in general, any data which can be packetized. 



In the embodiment of Figure 3, sender system 305 sends information 
including video data to a client 355 over channel 360. Video data 310 is 
15 captured, encoded and packetized by encoder/packetizer 320. The packets 
are next stored in storage medium 330 for subsequent delivery. 



Media data can be transmitted in response to a client request, or at a 
preset time for example. When it is time for the media to be transmitted, 

20 interleaver 350 adapts the conventional schedule (numerically sequential) 
into an adapted schedule. Interleaver 350 is a schedule adapter that, 
according to one embodiment, may be a hardware device well-known in the 
art, or it may, according to another embodiment, be a packet scheduler that 
consists of computer-readable code residing in a memory of sender system 

25 305 and executed by a processor residing at interleaver 350. The packet 
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scheduler can be programmed to perform as an interleaver and, as such, will 
be discussed herein as one embodiment of an interleaver. The adapted 
schedule is a function of the interleaver selected. The selection of the 
interleaver is performed by interleaver 350 based on characteristics of 
5 downstream channel 360 over which the video media packets are to be 
transmitted. 

The video distortion that is sometimes visible to a receiver is often a 
result of a burst loss or a delay of video media due to congestion or 

1 0 interference in the transmission process. If several consecutive packets of 
data are lost (a "burst" loss) or delayed for a significant amount of time, it 
becomes difficult to accurately decode the video media at its destination. For 
example, with MPEG (Moving Picture Experts Group) encoding, a P-frame is 
predicted from a preceding P-frame or l-frame, and a B-frame is predicted 

15 from two preceding P-frames or a preceding l-frame and P-frame. Thus, the 
burst loss can make it more difficult to decode and reconstruct the video 
media at the recipient or client end. 

In order to alleviate burst losses, the present invention includes a 
20 method for reordering the transmission of packets of media data using a 
reordering device such as interleaver 350. Interleaver 350 is a schedule 
adapter that may be selected from a collection of several types and/or 
configurations of interleavers. These interleavers include block, 
convolutional, switching or any type of interleaver that can be adaptable to a 
25 media streaming system, or, as discussed above, it can be a computer- 
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readable code that schedules packets and that may be programmed to 
perform as an interleaver. Each interleaver has a different configuration and 
is selected based on characteristics of channel 360. 



5 Figures 4A and 5A illustrate examples of two different configurations 

of block interleavers that can be used. In another embodiment of the present 
invention, a schedule adapter that can be used, similar in function to an 
interleaver, is a switching mechanism that switches alternatively among 
multiple channels in order to convert burst losses into isolated losses at the 
10 receiver end. Figure 8 illustrates one embodiment of a multi-channel 
switching adapter. 



Once the receiver/decoder 365 of Figure 3 receives a transmission, it 
de-interleaves and decodes it and sends decoded media 370 on to client 

15 355 for display. In one embodiment, receiver/decoder 365 also relays 
information about channel loss statistics to channel characteristics 
accumulator 375. Channel characteristics accumulator 375 can then 
accumulate the channel characteristics data and send the data back, on a 
predetermined periodic basis, to channel characteristics estimator/historical 

20 data retriever 380 at the sender's end. 



Still referring to Figure 3, the initial packets transmitted can be sent 
based on a standard or default schedule as determined by interleaver 
selector 385, or channel data retrieved from a historical data storage 390. 
25 Then as channel characteristics data becomes available from channel 
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characteristics accumulator 375 at a receiver end of a transmission, 
interleaver selector 385 can select an interleaver 350 with a configuration 
that would better reduce distortions affiliated with channel 360 based on 
channel characteristics. The interleaver selector 385 can choose an 
5 available interleaver that will produce the lowest distortion of all of the 
available interleaves, or it can choose an interleaver that satisfies a 
predetermined distortion threshold. This selection can be performed on a 
scheduled basis from channel characteristics accumulator 375. 



10 It is also possible that channel characteristics data, rather than being 

accumulated, can, according to one embodiment, be sent back on a per- 
packet basis. That is, as each packet is received and decoded, channel loss 
characteristics can be sent directly, in real time. This then allows real-time 
interleaver selection and reordering of transmission to accommodate the 

1 5 loss characteristics of channel 360 "on-the-fly". 



The average burst length (average number of consecutive packets 
lost), the average time between losses and the estimated time between 
losses are example statistics for determining the channel loss 

20 characteristics. Channel loss characteristics can then be used for 

determining an order in which to transmit the packets of data. According to 
one embodiment, the channel loss data includes an average packet loss 
rate and the number of times different loss events occur. For example, it 
includes the number of times an isolated packet loss occurred, the number of 

25 times sequential losses of packets occurred, the number of bursts of length 2 
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packets, the number of bursts of length 3 packets, the number of bursts of 
length 4 packets, etc., the pattern of the packets received and lost, the end- 
to-end delay in receipt of packets and the time between losses. 

5 In one embodiment, the receiver/decoder 365 sends information 

regarding which packets were received and which were lost to the channel 
characteristics accumulator 375. The sender's characteristics 
estimator/historical data retriever 380, upon receipt of the data from the 
client's channel characteristics accumulator 375, can then determine the 
10 pertinent statistics from the information received. The information can then 
be stored in historical data storage media 390 for future retrieval. 

In another embodiment, distortion can be simulated for varying 
combinations of channel losses and interleavers. For example, burst losses 

15 of 3 packets could be simulated with a (3,4) interleaver, a (3,5) interleaver, a 
4,3) interleaver, etc. and the results queried for the combination that 
produced the least distortion for a 3-packet loss. Likewise, the same 
information could be collected for burst losses of 2 packets, 4 packets, 5 
packets, etc., for all available interleavers. From this information, a table 

20 containing the results can be generated and stored. This table can then be 
used for predicting the minimum distortion in media received for given 
channel losses and a given interleaver. Thus, the selection of an interleaver 
by interleaver selector 385 could be a matter of consulting the table. Another 
variable that could be considered in the distortion prediction is the order of 

25 the frames being transmitted in each packet and from packet to packet. 
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There may be times in which there is no feedback of channel 
characteristics, such as when the media is broadcast to many clients. In 
such instances, historical data for given channels can be relied upon for 
5 estimating the packet transmission order. 



Once characteristics estimator/historical data retriever 380 identifies 
the channel characteristics between the sender system 305 and the client 
355, it examines the various eligible interleavers 350 and determines the 
1 0 one that will provide minimum expected total distortion as seen by the client 
355 for the channel being used while satisfying any delay constraints that 
might exist. In other words, the size of a block interleaver might cause an 
unacceptable delay in transmission if it is too large. 



1 5 Another possible consideration in the selection of the interleaver is 

the original order of the encoded media packets in terms of number of P- 
frames and/or B-frames between l-frames as in MPEG encoding. It may not 
be desirous to reorder the transmission so that, for example, all l-frames 
were sent consecutively. If this were the case, a burst loss of several I- 

20 frames could increase the distortion. For example, if the packets are coded 
in a periodic manner, such as IPPPIPPPIPPP, then isolated losses of every 
fourth frame would result in loss of all l-frames. In this instance, it may be 
preferable to have all the losses together as in a burst loss. Therefore, it is 
desirous to transmit the packets in an order that would preserve as many I- 

25 frames as possible, thus resulting in reduced distortion. 
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An important point is that the choice of schedule adapting strategy 
depends on the specific media coding mode per frame (e.g., MPEG I, P, B), 
the channel conditions, delay constraint (if necessary), and the specific 
5 media content. For example, for a given IPB coding pattern, channel 
conditions and delay, the choice of interleaver will still vary depending on 
whether the media content is largely static (e.g., media-conference) or highly 
dynamic (e.g., basketball game). This is because the same packet loss 
pattern will produce different resulting distortions for different media 

10 sequences. Therefore, appropriate models that can accurately predict the 
resulting distortion for a given media sequence and different packet loss 
patterns should be used to select the best interleaver. Models for accurately 
predicting the distortion for a given media sequence and a given packet loss 
pattern are known in the art. Such models can be used to estimate the 

1 5 distortion for different packet loss patterns that may result from the use of 
different possible interleavers. The interleaver that achieves the lowest 
expected distortion and, thus, the highest expected quality, can be chosen 
as the preferred interleaver to use for a given channel and a given media. 

20 Specifically, given knowledge of the encoded media packet order, 

channel conditions and a delay constraint (e.g., 400 ms) the sender 305 
determines which of the possible interleavers 350 would be expected to 
produce the least expected distortion in decoded media 370 at the client 
355. This may be a single selection of an interleaver 350 for a given 

25 transmission when channel 360 exhibits stable characteristics, or it may 
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require dynamic changing of interleaver 350 based on real-time feedback 
from the recipient client device 365 regarding the loss characteristics of 
channel 360. 

5 There are situations in which characteristics of channel 360 vary as a 

function of time. For example, as the congestion increases, packet loss rates 
can increase. Also, for a wireless link, as the distance between sender 305 
and client 355 increases, the received signal-to-noise ratio (SNR) 
decreases. Another example of variation with time might be that, as more 
1 0 wireless transmitters try to use the same bandwidth, additional interference 
occurs (again, the SNR goes down). 

In one embodiment of the present invention, receiver/decoder 365 
can gather the applicable statistics and send them back via channel 

15 characteristics accumulator 375 to sender's channel characteristics 
estimator/historical data retriever 380. Channel characteristics 
estimator/historical data retriever 380 can then reassess the appropriate 
interleaver and perhaps select a different one. A different interleaver would 
alter the schedule for transmitting packets in order to improve the quality of 

20 decoded media 370 received by client 355. 

Figure 4A is an illustration of a block interleaver 400a, in accordance 
with one embodiment of the present invention. Block interleaver 400a is one 
example of an interleaver configuration that can be used at a sender (e.g., 
25 interleaver 350 at sender 305 of Figure 3) to interleave packets of media 
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data before transmission. Numerically sequential packets are first read into 
the interleaver in horizontal rows as shown by arrow 410, with each row 
accommodating n packets. Packets are transmitted in columns as indicated 
by arrow 420 as soon as d rows of packets are filled. Here n is referred to as 
5 the block size and d is the interleaving depth of interleaver 400a. An 
interleaver can be identified by (n,d), so that interleaver 400a is a (3,4) 
interleaver. 

Figure 4B illustrates a stream of media packets, as transmitted from 
10 interleaver 400a, having a burst loss of three packets, 7, 10 and 2. For 
purposes of discussion it is assumed that a packet contains a frame of 
media, although in practice this may not be the case. Once these packets 
arrive at their destination receiver and are de-interleaved and decoded, the 
frame contained in packet 1 can be used to predict the lost frame from packet 
15 2 as indicated by arrow 450. Likewise, frame 6 is available to predict frame 
7, shown by arrow 440 and frame 9 can predict frame 10, shown by arrow 
430. Thus the burst loss of three packets is converted to three separate and 
isolated losses by use of interleaver 400a. Since the burst loss now affects 
separated packets instead of successive packets, the distortion can be 
20 expected to be lower, resulting in a higher quality transmission. A (3,4) 

interleaver can turn burst losses of up to four packets into effective isolated 
losses with respect to the original (pre-interleaved) order. 

The effectiveness of an interleaver depends on the block size n , the 
25 interleaving depth d, and the characteristics of the channel over which the 
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packets are transmitted. A larger interleaver is more effective in that it can 
convert a longer burst loss into isolated losses or increase the separation of 
the converted isolated losses. However, this is at the cost of higher latency. 
At the client, an interleaved packet received cannot be used until all the 
5 packets it depends on are received. Therefore, a trade-off exists between 
the effectiveness in permuting the packets and the latency. This tradeoff 
may, according to one embodiment of the present invention, be considered 
in selecting an appropriate interleaver for adapting the schedule of packets 
to be transmitted over a given channel. 

10 

Another important consideration in the selection of an appropriate 
interleaver is the sequence of frames in the stream of packets to be 
reordered. For example, if a stream of packets, each containing a frame, 
was of a sequence in which every 5 th packet contained an l-frame, it would 

15 not be desirable to use an interleaver in which n is five. An interleaver such 
as a (5,4) interleaver would, in such a case, transmit four l-frames 
sequentially. Thus, a burst loss that might occur could have the potential of 
losing up to four consecutive l-frames. Such a loss could result in increased 
distortion. Therefore, the sequence of frames in the incoming media stream 

20 is an important consideration in the selection of aninterleaver or other device 
with which to reorder the sequence of media packet transmission. 

Figure 5A is an illustration of a (3,5) block interleaver 500a that may 
be selectable by interleaver selector 385 if Figure 3, in accordance with one 
25 embodiment of the present invention. Interleaver 500a has a block size of 
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three packets and an interleaver depth of five. Thus, in this example, media 
packets 1 - 15 are read in horizontally into rows of three packets and then 
read out vertically in columns of five. 

5 Figure 5B illustrates a stream of media packets as scheduled by block 

interleaver 500a having (n,d) of (3,5) as is illustrated in Figure 5A, in 
accordance with one embodiment of the present invention. It can be seen 
that a burst loss of up to five packets can be changed to effective isolated 
losses by the (3,5) interleaver, but there is a longer delay time for reading in 

10 fifteen packets into interleaver 500a than to read in twelve packets into 

interleaver 400a of Figure 4A. Thus, there becomes a trade-off decision to 
be factored into the selection of an interleaver to accommodate channel 
characteristics. If a given channel rarely sees a burst loss greater than three 
packets in length, (3,4) interleaver 400a would be appropriate, but if losses 

15 of five packets in length are common, interleaver 500a might be preferred. 
However, there are other considerations as enumerated in the foregoing 
discussion in conjunction with Figures 4A and 4B. 

System for Live Video 

20 

Figure 6 is a block diagram 600 of sender system 670 for transmitting 
live video data for real-time transmission, according to one embodiment of 
the present invention. Here a camera 610 records an image 605 and the 
recorded video is fed into encoder/packetizer 320 for encoding and 
25 packetizing in preparation for transmission over channel 650. 
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Schedule adapter 620 then adapts the order of the packets based on 
information it has regarding channel characteristics. Schedule adapter 620 
can be a collection of several configurations of interleavers, each with a 
5 different configuration, selectable based on characteristics of channel 650. 
Figures 4A and 5A illustrate examples of two different configurations of block 
interleavers that can be used. In another embodiment of the present 
invention, schedule adapter 620 is a switching mechanism that switches 
alternatively among multiple channels in order to convert burst losses into 
10 isolated losses at the receiver end. Figure 8 illustrates such a multi-channel 
switching adapter. 

Still referring to Figure 6, the initial packets transmitted can be sent 
based on a standard or default schedule as determined by adapter selector 

1 5 385, or channel data retrieved from a historical data storage 390. Then as 
channel characteristics data becomes available from channel characteristics 
accumulator 375 at a receiver end of a transmission, adapter selector 385 
may select a schedule adapter 620 with a configuration that would better 
reduce distortions affiliated with channel 650 based on channel 

20 characteristics. This selection can be performed dynamically as channel 
loss data is sent from recipient device 630, or on a scheduled basis from 
channel characteristics accumulator 375. 

Once the receiver/decoder 630 receives a transmission, it de- 
25 interleaves it, decodes it and sends decoded media 640 on to a client for 
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display. In one embodiment, receiver/decoder 630 sends channel loss 
information, on a packet-by-packet basis, back to channel characteristics 
estimator/historical data retriever 380. In one embodiment, receiver/decoder 
630 also relays information about channel loss statistics to channel 
5 characteristics accumulator 375. The channel characteristics accumulator 
375 can then send the information back to the sender's channel 
characteristics estimator/historical data retriever 380 at the sender's end. 
According to one embodiment, the information accumulated may include an 
average packet loss rate, the number of times different loss events occur, 
1 0 and the average time between loss events. For example, it may include the 
number of times an isolated loss occurred, the number of bursts of length 2 
packets, the number of bursts of length 3 packets, the number of bursts of 
length 4 packets, etc., as well as the time between losses and the end-to-end 
delay time. 

15 

In another embodiment, the receiver/decoder 630 sends information 
regarding which packets were received and which were lost to the channel 
characteristics accumulator 375. The sender's characteristics 
estimator/historical data retriever 380, upon receipt of the data from the 
20 client's channel characteristics accumulator 375, can then determine the 
pertinent statistics from the information received. The information can then 
be stored in historical data storage media 390 for future retrieval. 



Once characteristics estimator/historical data retriever 380 identifies 
25 the channel characteristics between a sender and a client, it examines the 
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various eligible schedule adapters 620 and determines the one that will 
provide minimum expected total distortion as seen by the client for the 
channel being used, while satisfying any delay constraint (if a delay 
constraint exists). Specifically, given knowledge of the channel conditions 
5 and a delay constraint (e.g., 400 ms) the sender determines which of the 
possible schedule adapters 620 would produce the least expected distortion 
in decoded media 640 at the client. 

Mid-network Re-scheduling 

1 0 Figure 7 is a block diagram of a system for adapting the scheduling of 

media packets received from one channel and transmitting them according 
to the adapted schedule over another channel, in accordance with one 
embodiment of the present invention. In the present embodiment, a sender 
710 transmits a media stream 705 in a conventional manner, in that the 

1 5 media 705 is encoded and packetized at encoder/packetizer at some point 
320 and stored in storage media 330. At an appropriate time for 
transmission, media packets are transmitted across channel A 720 in a 
numerically sequential stream. In another embodiment, the sender may be 
performing real-time capturing, encoding and streaming. 

20 

Sender 725 receives the media data and adapts the scheduling with 
schedule adapter 730 to reduce distortion across channel B740 based on 
estimated channel characteristics of channel B 740. Schedule adapter 730 
may be a collection of several configurations of interleavers, each with a 
25 different configuration, selectable based on characteristics of channel 740. 
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Figures 4A and 5A illustrate examples of two different configurations of block 
interleavers that can be used. In another embodiment of the present 
invention, schedule adapter 730 may be a switching mechanism that 
switches alternatively among multiple channels in order to convert burst 
5 losses into isolated losses at the receiver end. Figure 8 illustrates such a 
multi-channel switching adapter 

Once receiver/decoder 750 of Figure 7 receives a transmission, it can 
de-interleave and decode it, and send decoded media 760 on to client 745 
for display. In one embodiment, receiver/decoder 750 also relays 
information about channel loss statistics to channel characteristics 
accumulator 375. The channel characteristics accumulator 375 can then 
send the information back to the channel characteristics estimator/historical 
data retriever 380 at the sender 725. According to one embodiment, the 
information accumulated can include an average packet loss rate and the 
number of times different loss events occur. For example, it may include the 
number of times an isolated loss occurred, the number of bursts of length 2 
packets, the number of bursts of length 3 packets, the number of bursts of 
length 4 packets, etc, the time each packet is received and the end-to-end 
delay times for each packet. 

In another embodiment, the receiver/decoder 750 can send 
information regarding which packets were received and which were lost to 
the channel characteristics accumulator 375. The characteristics 
25 estimator/historical data retriever 380, upon receipt of the data from the 
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client's channel characteristics accumulator 375, can then determine the 
pertinent statistics from the information received. The information can then 
be stored in historical data storage media 390 for future retrieval. The data 
can also be used dynamically, on a real time basis, for adjusting the selected 
5 schedule adapter in order to reorder the transmission sequence of media 
packets to that which will reduce distortion that might be associated with 
downstream channel 740. 

Once the characteristics estimator/historical data retriever 380 of 
1 0 sender 725 identifies the channel characteristics between a sender and a 
client, it examines the various eligible schedule adapters 730 and 
determines the one that will provide minimum expected total distortion as 
seen by the client for the channel being used, while satisfying any delay 
constraint (if a delay constraint exists). Specifically, given knowledge of the 
15 channel conditions and a delay constraint (e.g., 400 ms) the sender 

determines which of the possible schedule adapters 730 would produce the 
least expected distortion in decoded media 760 at the client. 

Mid-network Re-scheduling of Scheduled Streaming 

20 In an alternative embodiment, sender 710 in Figure 7 may already be 

performing appropriate scheduling for channel A. For example, sender 710 
may be operating as described by sender system 305 in Figure 3 or sender 
system 660 in Figure 6. In this case, the mid-network sender 725 can 
reschedule the already scheduled packet transmission to further reduce any 

25 distortion produced in the two channels. For example, sender 710, 
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operating alone, would have to select a schedule that would best reduce the 
distortion for the end-to-end channel corresponding to the concatenation of 
channels 720 and 740. However, by placing the mid-network re-scheduler 
in between, sender 710 only needs to be concerned with the distortion 
5 reduction associated with channel 720 and re-scheduler 725 can then select 
the interleaver that best reduces the distortion associated with channel 740. 
Specifically, sender 710 controls the scheduling for channel 720 and sender 
725 re-schedules as needed to a preferred schedule for channel 740. This 
can lead to improved performance since each channel is considered 
1 0 separately for improving distortion due to packet losses (as opposed to 
concatenated losses) and can provide lower end-to-end delay times. 



Scheduling Across Multiple Channels 

Figure 8 illustrates a schedule adapter 800 utilizing multiple channels, 
15 according to one embodiment of the present invention. Schedule adapter 
800 has available to it four different channels 830, 840, 850 and 860 
according to the present embodiment. In another embodiment there can be 
three, five, six, or any number of channels that might be made available to 
schedule adapter 800. 

20 

Wherein an interleaver, such as interleaver 400a in Figure 4A, changes 
the order with respect to time in which packets are sent out over a single 
channel, schedule adapter 800 receives numerically sequential packets 810 
from a conventional encoder/packetizer (e.g., encoder/packetizer 320 of 
25 Figure 3) and forwards them in the same order with respect to time, but over 
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different channels by way of switching mechanism 820. For example, channel 
830 may receive packet 1, channel 840, packet 2, channel 850, packet 3, 
channel 860, packet 4, then, 830 may next receive packet 5, etc. The result 
would be a media stream that would look like that of (3,4) interleaver 400a of 
5 Figure 4A in accordance with one embodiment. 

Still referring to Figure 8, the packets would potentially be received at 
receiver/decoder 870 in the same numerically sequential order as that in 
which they were originally scheduled. In another embodiment, 

10 receiver/decoder 870 is downstream of a switch similar to switch 820 that 
can mirror the movement of switch 820 for receiving the packets in the 
channel order in which they were sent. Receiver/decoder 870 can then be 
programmed to check the order and re-sequence any out-of-sequence 
packets that may have been delayed due to some unforeseen characteristics 

15 of the channel over which they were streamed. 

Consider that changing the order of a stream of packets using an 
interleaver such as interleaver 350 of Figure 3 might be thought of as re- 
scheduling "over time", and changing the order using multiple channels as 
20 illustrated in Figure 8 might be thought of as re-scheduling "over space". 
Although each has been treated as a separate embodiment, now consider 
an embodiment in which the packets are scheduled across both time and 
space. In one embodiment, this may be accomplished through 
concatenation of an interleaver and multiple channels. Such a process 
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would result in a more random sequence of packets that could further reduce 
distortion resulting from packet loss, thus improving performance. 



In yet another embodiment the scheduling of packets across both 
5 space and time may be accomplished as an integrated process in which the 
packets are scheduled randomly across multiple channels. In this 
embodiment, the randomizing of the packet order could be even further 
improved and, resultantly, so could channel performance be further 
improved. 

10 

Figure 9 is a flowchart of a method 900 for converting burst losses into 
isolated losses, in accordance with one embodiment of the present 
invention. Although specific steps are disclosed in flowchart 900, such steps 
are exemplary. That is, embodiments of the present invention are well suited 

1 5 to performing various other steps or variations of the steps recited in 

flowchart 900. All of, or a portion of, the methods described by flowchart 900 
can be implemented using computer-readable and computer-executable 
instructions which reside, for example, in computer-usable media of one or 
more computer systems. Generally, flowchart 900 can be implemented by 

20 system 300 of Figure 3. 



In step 910, in the present embodiment, media data enters an 
encoder/packetizer (e.g., encoder/packetizer 320 of Figure 3) where it is 
compressed, encoded and packetized for future transmission. It can then be 
25 further processed according to the following steps, or stored in a storage 
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area to await scheduling and transmission when requested from a client 
device. 



In step 920 of method 900, in accordance with one embodiment of the 
5 present invention, an interleaver is selected by an interleaver selector. The 
interleaver can be a collection of several configurations of interleavers, each 
with a different configuration, selectable based on characteristics of the 
channel over which transmission will occur. The interleaver selector selects 
an interleaver from those available that will provide the lowest amount of 

10 distortion or that will satisfy a predetermined distortion threshold. Figures 4A 
and 5A illustrate examples of two different configurations of block 
interleavers that can be used. In another embodiment of the present 
invention, other configurations and/or types of interleavers, (e.g., 
convolutional) or schedule adapters can be used. One example of a 

15 schedule adapter that can be used, similar in function to an interleaver, is a 
switching mechanism that switches alternatively among multiple channels in 
order to convert burst losses into effective isolated losses at the receiver end. 
Figure 8 illustrates such a multi-channel switching adapter. 

20 In step 930, the selected interleaver then adapts the conventional 

schedule (numerically sequential) into an adapted or reordered schedule 
based on characteristics of the channel over which the media is to be 
transmitted. The adapted schedule is selected to convert burst losses that 
might occur into isolated losses as received, so as to reduce distortion in the 

25 media at a receiver end. The channel characteristics can be determined by 
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a channel characteristics estimator/historical data retriever based on 
channel data received from the receiver end. 



In step 940 of Figure 9, a receiver/decoder receives, de-interleaves 
5 and decodes the media packets, sends the data to the client and sends 

information regarding packet losses to a channel characteristics accumulator 
and/or back to the sender system on a real-time basis in accordance with 
one embodiment. The channel characteristics accumulator can then send 
the information back to the sender's channel characteristics 
1 0 estimator/historical data retriever at the sender's end. 



According to one embodiment, the information accumulated may 
include an average packet loss rate, the number of times different loss 
events occur, and the average time between loss events. For example, it 

15 may include the number of times an isolated loss occurred, the number of 
bursts of length 2 packets, the number of bursts of length 3 packets, the 
number of bursts of length 4 packets, etc., the end-to-end delay of each 
packet and the time between losses. In another embodiment, the channel 
characteristics accumulator may accumulate information regarding which 

20 packets were received and which were lost. 



In step 950, the channel characteristics accumulator sends the 
accumulated information to the sender's characteristics estimator/historical 
data retriever, according to one embodiment. Upon receipt of the channel 
25 characteristics data, the sender's characteristics estimator/historical data 
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retriever determines the pertinent statistics from the information received. 
The information can then be stored in a historical data storage media for 
future retrieval. The process continues until all data has been transmitted at 
which time method 900 is exited. 

5 

In summary, in its various embodiments, the present invention 
provides methods and systems thereof for reducing distortion and improving 
the quality of media received by determining and implementing a strategy for 
reordering the transmission across media data packets in order to reduce 

1 0 the effects of media packet losses. The features of the present invention are 
well suited for, but not limited to, media transmission over channels with 
significant losses of sequential packets (burst losses). According to the 
various embodiments of the present invention, the quality of media data 
received can be significantly improved by isolating packets lost so that they 

15 can be predicted by a packet that was recorded relatively close in time to the 
lost packet. 

Embodiments of the present invention are thus described. While the 
present invention has been described in particular embodiments, it should 
20 be appreciated that the present invention should not be construed as limited 
by such embodiments, but rather construed according to the following 
claims. 
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